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Abstract 

We present an explicit product formula for the spherical functions of the compact 
Gelfand pairs (G, Ki) = {SU{p -I- q), SU(p) x SU(q)) with p > 2q, which can be consid¬ 
ered as the elementary spherical functions of one-dimensional A-type for the Hermitian 
symmetric spaces G/K with K = S{U{p) x U{q)). Due to results of Heckman, they can 
be expressed in terms of Heckman-Opdam Jacobi polynomials of type BCq with spe¬ 
cific half-integer multiplicities. By analytic continuation with respect to the multiplicity 
parameters we obtain positive product formulas for the extensions of these spherical func¬ 
tions as well as associated compact and commutative hypergroup structures parametrized 
by real p €]2q— 1, oo[. We also obtain explicit product formulas for the involved continu¬ 
ous two-parameter family of Heckman-Opdam Jacobi polynomials with regular, but not 
necessarily positive multiplicities. The results of this paper extend well known results for 
the disk convolutions for q = 1 to higher rank. 

Key words: Hypergeometric functions associated with root systems, Heckman-Opdam the¬ 
ory, Jacobi polynomials, disk hypergroups, positive product formulas, compact Grassmann 
manifolds, spherical functions. 
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1 Introduction 

It is well-known that the spherical functions of Riemannian symmetric spaces of compact 
type can be considered as Heckman-Opdam polynomials, which are the polynomial vari¬ 
ants of Heckman-Opdam hypergeometric functions. In particular, the spherical functions of 
Grassmann manifolds SU{p + q,¥)/S{U{p,¥) x U{q,¥)) with p > q and F = M,C,EI can be 
realized as Heckman-Opdam polynomials of type BCq with certain mulitplicities given by 
the root data, namely 

k = {d{p — q)/2, (d — l)/2, d/2) with d = dim^F G {1, 2,4}, 

corresponding to the short, long and middle roots, respectively. We refer to m, 0 and 
inii for the foundations of Heckman-Opdam theory, and to |RR] for the compact Grassmann 
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case. Recall that the spherical functions of a Gelfand pair (G, K) can be characterized as 
the continnous, R-biinvariant functions on G satisfying the product formula 

= [ (p{gkh)dk {g,hGG) (1.1) 

Jk 

where dk is the normalized Haar measure on K. In [RR] , the product formula for the spherical 
functions of SU{p + q,¥)/S{U{p,F) x U{q,¥)), considered as functions on the fundamental 
alcove 

Ag = {t£U^ : TT/2>ti>t2>...>tq> 0}, (1.2) 

was extended by analytic interpolation to all real parameters p G]2q — l,oo[. This led to 
positive product formulas for the associated RC-Heckman-Opdam polynomials as well as 
commutative hypergroups structures on Aq with the Heckman-Opdam polynomials as char¬ 
acters. For a background on hypergroups see [J], where hypergroups are called convos. 
For the non-compact Grassmannians over M, C and H, similar constructions had previ¬ 
ously been carried ont in [R 2j . In the case F = C, where the non-compact Grassmannian 
SU{p,q)/S{U{p) xU{q)) is a Hermitian symmetric space, the approach of |R2| was extended 
to the spherical functions of U(j),q)/U{p) x SU{q) in [V3| . Here the analysis was based on 
the decisive fact that the spherical functions of this (non-symmetric) space are intimately 
related to the generalized spherical functions of the symmetric space SU{p, q)/S{U(p) xU{q)) 
of one-dimensional RT-type. These in turn can be expressed in terms of Heckman-Opdam 
hypergeometric functions according to m, Section 5. 

In the present paper, we consider the pair {G,Ki) with G = SU{p + q) and Ki = 
SU{p) X SU{q) (over F = C), where we assume p > q. In this case, (G, Ki) is a Gelfand pair. 
By a decomposition of Cartan type, we identify the spherical functions of GjKi as functions 
on the compact cone 

Xq := {{zri,r 2 ,... ,rg) : 0 < ri < ... < 1, z G T} C C x 

with the torus T := {z G C : \z\ = 1}. Similar as in the non-compact case [V3] . the spherical 
functions of GjKi can be considered as elementary spherical functions of RT-type xi (^ ^ ^) 
for the Hermitian symmetric space G/RT with K = S{U{p) x U{q)). According to the results 
of Heckman ESI, they can be expressed in terms of of Heckman-Opdam polynomials of type 
BGq, depending on the integral parameter p > q and the spectral parameter I G Z. Under 
the stronger requirement p > 2q, we proceed similar as in [RR] and write down the product 
formula for the spherical functions of GjKi as product formulas on Xg. 

In Section 4 we then extend this formula to a product formula on Xq for a continuous 
range of parameters p g] 2(7 — 1, oo[ by means of Carlson’s theorem, a principle of analytic con¬ 
tinuation. In particular, we obtain a continuous family of associated commutative hypergroup 
structures on Xg. We determine the dual spaces and Haar measures of these hypergroups. 
For (7 = 1, the space Xq is the complex unit disk, and the associated hypergroups are the 
well-known disk hypergroups studied in and m, where the associated product 

formulas are based on the work of Koornwinder |K2| . 

For each realp g] 2(7—1, oo[, the associated hypergroup structure on Xg contains a compact 
subgroup isomorphic to the one-dimensional torus (TT, •). Moreover, the quotient Xq/T can 
be identified with the alcove Aq and carries associated quotient convolution structures; see 
[J] and m for the general background. In this way we in particular recover the above- 
mentioned hypergroup structures of [RR] on Ag for F = C. More generally, we obtain 
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in Section 5 explicit continuous product formulas and convolution structures on Aq for all 
Jacobi polynomials of type BCq with multiplicities 

k = {ki,k 2 , ks) = {p-q- 1, 1/2 + 1, 1 ) 

with p G]2q — l,oo] and I E M. Unfortunately, for general I / 0, the positivity of these 
product formulas remains open. 

Acknowledgement. It is a pleasure to thank Maarten van Pruijssen for valuable hints 
concerning Gelfand pairs. 


2 Preliminaries 


We start our considerations with the compact Grassmann manifolds G/K over C, where 
G = SU{p + q) and K = S{U{p) x U{q)) with p > q > 1- From the Gartan decomposition 
of G (see [RR| or Theorem VII. 8.6 of [Hel] ) it follows that a system of representatives of the 
double coset space GjfK is given by the matrices 


at 



0 

cost 

sint 



t E Aq 


with Aq as in here t is the q x q-diagonal matrix with the components of t as entries, 

and cost, sint are understood componentwise. We recall from [RR] how the double coset 
representatives can be determined explicitly: 

For X E Mq{C) denote by asing{X) = ■\/spec(V*A) = (cJi ... ,aq) E the vector of 
singular values of X, decreasingly ordered by size. Write <7 E G in (p x g)-block notation as 


g = 


(A{g) Big)\ 
[G{g) D{g))^ 


and suppose that g E KbfK. Then 


( 2 . 1 ) 


t = arccosia sing{D{g))). (2.2) 

By [RRj, the spherical functions of G/K are given by Heckman-Opdam polynomials, as 
follows: Gonsider M”? with the standard inner product (., .) and denote by FsCq (■^j k\ t) the 
Heckman-Opdam hypergeometric function associated with the root system 

R = 2BGq = {±2ej, ±4ej, 1 < i < g} U {±2ei ± 2ej : 1 < i < j < q} C 

and with multiplicity parameter k = {ka)aeR, in the notion of |R2| . Recall that there exists an 
open regular set Tbq(A, k] .) exists in a suitable tubular neighbourhood 

of C C^. We also write k = (fei, ^ 2 , /ca) where ki,k 2 , k^ belong to the short, long and middle 
roots, respectively. Fix the positive subsystem i?_|_ = {2ej,4ej,2ej ± 2ej '■ 1 < i < j < q}- 
Writing a'^ := 2a/{a, a), the associated set of dominant weights is 

P+ := {A E M'? : (A,a^) E Z+ Va E R+} = {A E 2Z!(_ : Ai > A 2 > ... > Xq}. (2.3) 

Notice that our normalization of root sytems and multiplicities is in accordance with [HSj . 
m but differs from the “geometric” notion of [RRj, where both are rescaled by a factor 2. 
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We consider the renormalized Heckman-Opdam polynomials associated with R and k in 
trigonometric notion as in [EE], which are defined by 

Rx{k;t) = FBCgi>^ +p{k),k;it), A G P+. (2.4) 

Here ^ 

Pik) = ^ k»<^ 

aeR+ 

and 

gM — TT ^((A, + 2^«/2) TT r((/?(A:), + 2^0/2 + 

+ + M J1+ r((/j(A:),aV) + ife„/2) 

is the generalized c-function, with the convention ka /2 ■= 0 a/2 ^ R. The Heckman- 

Opdam polynomials are holomorphic on Indeed, FsCqi^ + p{k)-, A:; •) is holomorphic on 
all of if and only if A G P+, see |HSj . 

According to Theorem 4.3. of M, the spherical functions of G/K = SU{p+q)/S{U{p) x 
U{q)) can be considered as trigonometric polynomials on the alcove Aq C and are given 
by 

ipx{t) = Rx{k-,t), A G P+ 

with the multiplicity 

k = {ki,k2,k3) = (p-g, 1/2,1). 

For fixed g > 1 and p > 2q, the product formula (II.ip for the (px was written in |RR] as a 
product formula on Aq depending on p in a way which allowed extension to all real parameters 
p > 2g — 1 by analytic continuation; see Theorem 4.4. of [EE]- 

3 Spherical functions of {SU{p + q), SU{p) x SU{q)) and their 
product formula 

Let us turn to the pair (G, Ki) := {SU{p + g), SU{p) x SU{q)), where we assume p > q. In 
this case, {G,Ki) is a Gelfand pair according to the classification of [Krj . In this section, we 
derive an explicit product formula for the spherical functions of {G, Ki). First, we determine 
a system of representatives for the double coset space G//K\. For this, consider the compact 
set 

Xq := {x = {zri,r 2 ,..., r^) : z G T, G M with 0 < n < ... < < 1} C C x 

If g = 1, this is just the closed unit disc in C. For g > 2, the set Xq can be interpreted as a 
cone of real dimension g + 1 with Xq-i as basis set. 

For X = {zri,r 2 ,... ,rq) G Xq we write 

X = [r, z] with r = (ri,..., rg). 

Note that the phase factor z G T is arbitrary if ri = 0. Now consider the alcove Aq. We 
define an equivalence relation on Ag x T via 

{t, z) ~ {t', z') :<(=^ t = t' and z = z' if ti = t/ < — . 

Then the mapping Ag x T —>■ Xq , {t,z) [cost, 2 ;] induces a homeomorphism between the 

quotient space (Ag x T)/ ~ and the cone Xq. 


4 












3.1 Lemma. For z G T, let h{z) = diag{z, 1... , 1) G Mq(C). Then a set of representatives 
of GI jKi is given by the matrices bx,x G Xq with 


bx 


/Ip-q 0 

1 0 h{z~^) cost 

\ 0 sin t 



for X 


[cost, 2 ;] withtGAq. 


(3.1) 


In particular, the double coset space GjjKx is naturally homeomorphic with the cone Xq. 

Proof We first check that each double coset has a representative of the stated form. In fact, 
by the known Cartan decomposition of G with respect to K, each g G G can be written as 


9 = 


f ui 

VO 


Vl 


at 


U2 0 
0 V2 


with t ^ Aq and Ui ^ U{p), Vi £ U{q) satisfying det(«i) • det(ui) = 1. As U{q) = SU{q) xi Hq 
with Hq = {h{z) : z G T}, there are zi,Z2 £T such that vi := vih{zi)~^ G SU{q) and V2 '■ = 
h{z2)~^V2 G SU{q). For z,w £T, define H{w, z) = diag(r(;, 1 ,... , 1 , z, 1,.. . , 1 ) G Mp(C) with 
the entry 2; in position p + 1. Put := rti • Ff(2;i/2;2,2:2) and U2 ■= H{z2/zi, zi) ■ U2 £ U{p). 
Then ui,U2 £ SU(p) , and a short calculation in p x (7-blocks gives 


9 = 





0 

cos t 
sint 



(U2 0\ 

Vo V 2 J 


(ui 0 \ 

Vo vij 



0 

h{zi)~^h{z 2 )~^ cost 
smt 


0 

— sint 
h{zi)h{z 2 ) cos t ^ 


f U2 
VO 



Thus g £ KibxKi with x = [cos t, 2 : 12 : 2 ]. 

In order to show that the bx are contained in different double cosets for different x, we 
analyze how the parameter x depends on g £ G. Let 


9 = 


f ui 

Vo 




with Ui £ SU{p), Vi £ SU{q). Suppose that x = [r,z] with r = cost, t £ Aq. Using the 
{p X q)-block notation ()2.1I) . we obtain 


g 

D{g) = vih{z)rv 2 , X{D{g)) = ^ • • |A(T>(5'))|, 

i=l 

Here and throughout the paper, A denotes the usual determinant of a complex square matrix. 
Thus r = cost = (Jsing{D{g)). Further, 2 ; is uniquely determined by g exactly if ri / 0, 
which is equivalent to 0?=! U / 0. In this case, 2 ; = . Thus {t, 2 :) G Hg x T is uniquely 

determined by g up to the equivalence ~, which proves our statement. 

□ 


The proof of the above lemma reveals the following equivalence for g £ G: 

g £ KibxKi X = [r,z] with r = asing{D{g)), z = argA(D(g)), 
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with the argument arg : C —)• T dehned by 


arg z := — if z 7 ^ 0 , arg 0 := 1 . 
jzj 

We are now going to write the general product formula (ll.lh for spherical functions as a 
product formula on the cone Xg = (Ag x T)/ ~ . For x = [cost,zi], x' = [cost',z'] E Xg and 
iili-biinvariant / E C(G) we have to evaluate the integral 


/ 

JKi 


f{bxkbx') dk. 


Write A: = ( Q ^ j with u E SU{p), v E SU{q). Then {p x ( 7 )-block calculation gives 


bxkbr' = 


where 


With the block matrix 


* D{bxkbx 

D{bxkbx') = {0,smt) u ^ + h{z) cos tv h{z') cos ^ . 

^0 := E MpJC) 


this can be written as 

Dibxkbx’) = — sintcjgMCTosint' + h{zz') costv cost'. (3-2) 

Regarding /Ci-biinvariant functions / E C{G) as continuous functions on Xg, we have 

/ f{bxkbx')dk = / f {[asing{D{bxkbx')),axgA{D{bxkbx')])dk 

JKi JsU{p)xSU{q) 

with D{bxkbx') from ()3.2I) . Notice that CQuac E Mg{C) is the lower right q x g-block of u 
and is contained in the closure of the ball 

Bg := {w E Mg{C) : w*w < Ig}, 

where w*w < Ig means that Ig — w*w is positive semidefinite. We now assume that p > 2q 
and reduce the S'?7(p)-integration by means of Lemma 2.1 of [R2| . Notice first that for 
continuous g on Bg, 

/ g{a^uao)du = / g{aQuao)du, 

Jsu{p) Ju{p) 

where du denotes the normalized Haar measure in each case. Thus Lemma 2.1 of 


gives 


[ f{bxkbxi)dk = — j I f [[asing{—sintwsint'+ h{zz') costv cost') 
iKi I'^p JBq Jsu{q) 


where 


aigA{—sintwsin^ + h{zz') costv cost')]) A{Ig — w*w)^ ‘^‘’dvdw, (3.3) 

Kp = [ A{Iq — W*W)^~^'^ dw 


'Bo 


and dw means integration with respect to Lebesgue measure on Bg. After the substitution 
w I—)• h{zz')w, we hnally arrive at the following 
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3.2 Theorem. Suppose that p > 2q. Then the product formula for the spherical functions 
of the Gelfand pair {G,Ki), considered as functions on the cone Xq, can he written as 

<^([cos z])(^([cos t',/]) = — [ I (p[[asing{—sintwsinT + costvcos-^), 

JBg Jsu{q) 

zz' ■ arg A( — sintu) sint' + cos tucost')]) • A(/q — w*wY~‘^^ dvdw. 

We remark that for p = 2q — 1, a degenerate version of this integral formula may be 
written down by using the coordinates introduced in Section 3 of [E lj . 

We next turn to the classification of the spherical functions of Note first that 

K = Ki X H with H = {H^, z G T} where is the diagonal matrix with entries z in position 
p — q + 1, 1/zin position p + 1 and 1 else. Let y : iL —>• T be the homomorphism with kernel 
Ki and x{Hz) '■= z. Then the characters of K are given by the functions Xi(fe) = x{^)\ ^ € Z, 
and we have the following characterization: 

3.3 Lemma. For (p G C{G) the following properties are equivalent: 

(1) p is Ki-spherical, i.e., Ki-biinvariant with p{g)ip{h) = p{gkh)dk for all g,h ^ G. 

(2) p is an elementary spherical function for {G,K) of type xi for some I G Z, i.e. p is 
not identical zero and satisfies the twisted product formula 

Ti9)T{h) = [ p{gkh)xi{k)dk forallg^heG. (3.4) 

Jk 

3.4 Remark. We here adopt the notion of elementary spherical functions of type xi according 
to m- Each such function automatically satisfies (p{e) = 1 as well as the x;-bi-coinvariance 
condition 

p{kigk 2 ) = Xi{kik 2 )~^ ■ pig) for all g G G,ki,k 2 G K, (3.5) 

see Lemma 3.2. of m- The discussion of |HS] is based on a different, but equivalent 
definition of elementary spherical functions of iL-type (for the non-compact dual), which 
requires (j3.5p together with a system of invariant differential operators on sections in an 
associated homogeneous line bundle, see Definition 5.2.1 of |HSj . For the equivalence of 
definitions we refer to Theorem 3.2 of [S]. 

Proof of Lemma \3.3[ The proof of this result, which we expect to be well-known, can be 
carried out for instance precisely as in Lemma 2.3 of [V3j . □ 

For an arbitrary irreducible Hermitian symmetric space and its compact dual, the ele¬ 
mentary spherical functions of type xi can be written as modifications of Heckman-Opdam 
hyper geometric functions, see Section 5 of [HS| . in particular Theorem 5.2.2 and Corollary 
5.2.3., as well as [HQ| . In the compact case, they correspond to the y/-spherical represen¬ 
tations of G which were classified in |Sch] . To become explicit in the particular case of our 
compact symmetric spaces G/K = SU{p +q)/S{U{p) x U{q)), recall the set P+ of dominant 
weights from (j2.3l) as well as the renormalized Heckman-Opdam polynomials Rx of type 
BGq from (j2.4p . According to [HSj . the elementary spherical functions of {G,K) of type Xh 
considered as functions on Aq, are given by 

<? 

f I—)• J]][ cos^^lfj • Rx{k{p, q, l)',t), A G P+ , / G Z (3.6) 

j=i 
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with multiplicity parameters 

Hp^qJ) ■={p-q- KI,i /2 + I e z. 

Notice at this point that for FsCq, the set 

{k = (/ci, k 2 , k^) : Re/c 3 > 0, Re(fci + /C 2 ) > 0} 

is contained in (see Remark 4.4.3 of |HS| i. and so in particular the multiplicities k{p, q, 1) 
are regular. The associated p-function is 

Q Q 

p{k{p,q,l)) = {p-q+\l\ + +2'^{q- j)ej. 

j=i j=i 

We now turn to the spherical functions of {G,Ki), which we again consider as functions on 
the cone Xg. 

3.5 Theorem. Let {G, K) = {SU{p+q), S{U{p)xU{q))) and Ki = SU{p) x SU{q). Then the 
spherical functions of the Gelfand pair (G, Ki), considered as functions on Xg, are precisely 
given by 

q 

ip^)^i{[cost,z]) = z'' ■Y\_cos^''hj ■ R\{k{p,q,l)-,t), X e , I e Z (3.7) 

i=i 

with the multiplicity k{p, q,l) = {p — q — \l\, 5 + |^|, 1 ) G 

Proof. First observe that ^ is indeed well-defined as a function on Xg, because the right 
side is zero if ti = 7 r/ 2 , independently of 2 ; G T. Now suppose that <p : G —>■ C is spherical for 
{G,Ki). Then by Lemma [331 it is xrspherical for some I G Z. Consider x = [cosRz] G Xg 
and write bx = with an arbitrary square root of z. Then in view of (13.51) . 

ip{bx) = z’- ■ ifiat), 

where (p{at) is of the form ()3.6p . This proves the statement. □ 

3.6 Example (The rank-one case q = 1). Here G/K = SU{p + l)/S(U{p) x U{1)) = 
U{p+l)/U{p) and G//K is homeomorphic to the unit disk = {z G C : |z| < 1} = D. We 
shall identify the spherical functions ^ as the well-known disk polynomials on D introduced 
in [K2] . which are known to be the spherical functions of {U{p + 1),U{p)). 

We have R+ = {2,4} C M and = 2Z+. According to the example on p.89f of [Olj . 
FbCi ^' 1 0 be expressed as a 2 F 1 - (Gaussian) hypergeometric function. Consider the 
renormalized one-dimensional Jacobi polynomials 

i?^“’^^(x) = 2 ^ 1(0 +/3 + n-|-1, —n, a-|-1; (1 — x)/2) (x G M, n G Z+) (3.8) 

for a,fd > —1. Then the Heckman-Opdam polynomials associated with R = 2BGi and 
multiplicity k = {ki, ^ 2 ) can be written as 

R 2 n{k]t) = i?i"’^^(cos 2t) (n G Z+, t G [0, 7 r/ 2 ]) (3.9) 

with 

a = ki + k 2 - 1 / 2 , (3 = k 2 - 1 / 2 ; 
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c.f. equation (5.4) of [RRj . where a different scaling of roots and multiplicites is used. Writing 
r = cost, we have cos2t = 2r^ — 1. We thus obtain from Theorem 13.51 the well-known fact 
that the spherical functions of {G,K) are given, as functions on D, by the so-called disk 
polynomials 

^n,i{zr) = (2r^ - 1) {z eT, r e [0,1]) (3.10) 

with I ^ Z, n £ Z_|_. Moreover, the product formula of Theorem 13.21 becomes in this case 


with 


j{zr) ■ (pn,i{z's) = — [ (pn,i{zz'{rs - uza/i - t^a/i - s^)) • (1 - \w\‘^y ^ dw (3.11) 
J D 


Kr) — 


[ {1 - \w\^Y-^ dw = 
J D 


TT 


p-l 


This formula is well known; see for instance m, m- 


4 Convolution algebras on the cone Xq for a continuous family 
of multiplicities 

In this section, we extend the product formula for the spherical functions of (G^Ki) in The¬ 
orem [Q from integers p > 2g to a continuous range of parameters p £]2q — l,oo[. We 
show that for each p £]2q — l,oo[, the corresponding product formula induces a commuta¬ 
tive Banach algebra structure on the space of all bounded Borel measures on Xq and an 
associated commutative hypergroup structure. These hypergroups generalize the known disk 
hypergroups for q = 1 which were studied for instance in m, |Ka| : see also the monograph 

m- 

As ususal, the basis for analytic continuation will be Carlson’s theorem, which we reca¬ 
pitulate for the reader’s convenience from HU: 

4.1 Theorem. Let f be holomorphic in a neighbourhood of {z £ C : Re z > 0} satisfying 

f{z) = for some c < it. If f{z) = 0 for all nonnegative integers z, then f is 

identically zero on {Re 2 ; > 0}. 

It is now straightforward, but a bit nasty in detail to prove that the product formula of 
Theorem 13.21 can be extended analytically with respect to the variable p. For the necessary 
exponential bounds, one has to use that the coefficients of the Jacobi polynominals Px{k] t) = 
c{X+p{k), k)~^R\{k', t) are rational in the multiplicity k, see Par. 11 of [M]- As the arguments 
are the same as those in the proof of Theorem 4.1 of |R2| and Theorem 3.2 of [V3] . we skip 
the details. We obtain: 

4.2 Theorem. Let p £]2q — l,oo[, A £ P+, and I £ Z. Then the functions 

<1 

;([cos t,z\) = z^ ■ cos^^l tj ■ R\{k; t) 

on Xq with multiplicity k = {p — q — \l\, ^ J- |/|, 1) € satisfy the product formula 

i{[cost, z]) ■ X^ost', z']) = — / / i[[asing{—sintw sin-^ + cost V cost/), 

’ ’ JBq Jsu{q) 

zz ■ arg A(— sintwsm^ + cos t ucos) • A(/q — w*dvdw. 
for {t,z), {f ,z') £ AqXT. 
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The positive product formula in Theorem l4.2l for p G]2g— 1, oo[ leads to a continuous series 
of probability-preserving commutative convolution algebras on the cone Xq. In fact, similar 
to the noncompact case |V3] , we obtain commutative hypergroups structures on Xq with the 
(A G P+, Z G Z) as hypergroup characters. To start with, let us briefly recapitulate some 
notions from hypergroup theory from m, [BH]. 

4.3 Definition. A hypergroup is a locally compact Hausdorff space X with a weakly con¬ 
tinuous, associative and bilinear convolution * on the Banach space Mb{X) of all bounded 
regular Borel measures on X such that the following properties hold: 

(1) For all x,y G X, 5x * Sy is a compactly supported probability measure on X such that 
the support supp((52; * 6y) depends continuously on x, y with respect to the so-called 
Michael topology on the space of compact subsets of X (see [J] for details). 

(2) There exists a neutral element e G X with 6x * Se = Sg * 6x = Sx for all x G X. 

(3) There exists a continuous involution x ^ x on X such that e G supp((ia; * 5y) holds if 
and only if y = x, and such that {5x*5y)~ = 6y*dx, where for p G Mi,{X), the measure 

denotes the pushforward of p under the involution. 


Due to weak continuity and bilinearity, the convolution of bounded measures on a hyper¬ 
group is uniquely determined by the convolution of point measures. A hypergroup is called 
commutative if so is the convolution. We recall from [j] that for a Gelfand pair {G,K), the 
double coset space GjfK carries the structure of a commutative hypergroup in a natural 
way. For a commutative hypergroup X the dual space is defined by 

X = {ip G Cb(X) : (p^O, (p(x* y) := (4 * = p{x)<p{y) ^x,y G X]. 


Each commutative hypergroup (X, *) admits a (up to a multiplicative constant unique) Haar 
measure ux, which is characterized by the condition taxif) = ^x{fx) for all continuous, 
compactly supported / G Gc{X) and all x G X, where fx denotes the translate fx{y) = 

( 5 ,* 4 )(/). 

Now let p G]2q — 1, (X)[. Using the positive product formula of Theorem 14.21 we introduce 
the convolution of point measures on Xq by 

(<^[cost+] *p ^[cosf ,z'])U) ■= — I I f{[asing{-smtwsmt^ -h costv cost'), 

J Bq J SU(q) 

zzfo argA(—sintrcsint' -|- costvcost!)\)X{Iq — w*w)'^~'^'^dvdw (4.1) 


for / G Gb{Xq). 

4.4 Theorem. Let y > 1 he an integer and p G]2q — 1, oo[. Then the convolution *p defined 
in ^.1\ ) extends uniquely to a bilinear, weakly continuous, commutative convolution on the 
Banach space Mb{Xq). This convolution is also associative, and {Xq,*p) is a commutative 
hypergroup with [(1,... , 1), 1] as identity and with the involution [r, z] := [r, z]. A Haar 
measure of the hypergroup {Xq,*p) is given by 


dujp{[r, z]) = rj{l - r])P ( 

j=i 


— r|)^ drdz 


with the Lebesgue measure dr on and the normalized Haar measure dz on T. Finally, the 
dual space is given by 


{Xq,*p)^ = {ip{y.XGP+,lGZ}. 
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Note that iOp is the pushforward measure under the mapping (t, z) ^ [cost, z], AqXT ^ 
Xq of the measure 


g 

dojp{t, -2) = ]^ costj tj ■ (cos(2tj) — cos{2tj))‘^dtdz 

j=i 


on Aq X T. 

For the proof of Theorem 14.41 consider the measure cOp defined above. We start with the 
following observation: 

4.5 Lemma. The functions 

ifl, {XeP+,leZ) 

form an orthogonal basis of L‘^{Xq,u!p), and their C-span is dense in the space C{Xq) of 
continuous functions on Xq with respect to ||.||cxd- 

Proof. For k = k{p,q,l), the Heckman-Opdam polynomials R\{k;t) are orthogonal on the 
alcove Aq with respect to the weight 

dk{t)= |gi{«4>/2 _g-i{a,t>/2|2fc. 

a&R+ 

<? 

= const ■ n gin2p-2g+l^^. . 

n (cos(2tj) — cos(2tj))^ 

j=i i<i<j<q 

This immediately implies that the functions {t,z) ^([cos t, z]) are orthogonal on x T 

with respect to ujp. Let V denote the subspace of C{Xq) spanned by ^ : A G P+, / € Z}. 
Clearly, V is an algebra which is stable under complex conjugation. Further, V separates 
points on Xq, because the Rx{k; .) span the space of Weyl group invariant trigonometric 
polynomials. Hence by the Stone-Weierstrafi theorem, V is dense in C{Xq) with respect to 
11.1 loo- The rest is obvious. □ 

Proof of Theorem \4-4\ The statements are clear for integer values of p, where *p is just the 
convolution of the double coset hypergroup SU{p + q)//SU{p) x SU{q). ,^From the definition 
of the convolution for general p we see that (5[r+] *p (5[r'+'] is a probability measure and that 
the mapping ([r, z], [r', z']) i—>■ d\^r,z\ *p is weakly continuous. It is now standard (see [J]) 
to extend the convolution of point measures uniquely in a bilinear, weakly continuous way 
to a probability preserving convolution on Mf,{Xq). For commutativity and associativity, it 
suffices to consider point measures. Let [ri, Zi] ^ Xq , \ < i < 3. Then for f = Txi 
A G P+ and ^ G Z we have 

(^[r-1,^1] *P ^[r2,Z2])if) = /(h,p])/(h,^2]) = iS[r2,Z2] *P <^[ri,2i])(/) 

and in the same way. 


((^[ri,zi] *p ^[r 2 ,Z 2 ]) *P ^[r3,23])(/) {^[ri,zi] *p (^[r 2 , 22 ] *P ^^3,23]))(/)• 

By Lemma 14.51 both identities remain valid for arbitrary / G C{Xq). The remaining hyper¬ 
group axioms are immediate from the fact that the supports of convolution products of point 
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measures are independent of p. For the statement on the Haar measure, the argumentation 
is similar to [RR]. Notice first that by definitition of hypergroup translates, the identity 



for all X £ Xq. 


holds for f = I with arbitrary A G P+, / G Z. In view of Lemma [4.51 it extends to arbitrary 
/ G C{Xq), hence Wp is a Haar measure. Finally, it is clear that the ^ are hypergroup 
characters. There are no further ones, because the characters of a compact hypergroup are 
orthogonal with respect to its Haar measure. 

□ 


The hypergroups {Xg , *p) have a prominent subgroup. For this we recall that a closed, 
non-empty subset P C Xg is a subgroup if for all x,y £ H, 5x *p Sy is a point measure with 
support in H. It is clear from (14.11) that 


H := {[1,2;] = (z, 1,... ,1) : 2 ; G T} 

is a subgroup of (Xg, *p) which is isomorphic to the torus group T. 
The cosets 

x*pH := [J supp {5x *p dy), x G Xg 
y&H 

form a disjoint decomposition of Xg, and the quotient 

Xq/H := {x *pH : x£ Xg} 


is again a locally compact Hausdorff space with respect to the quotient topology, c.f. Section 
10.3. of [J]. Moreover, 

{6x*,H*pSy,^H){f):= [ f{z*pH)d{6x*p6y){z), x,y£Xq,f£a{XqlH) (4.2) 

Jx 

establishes a well-defined quotient convolution and an associated commutative quotient hy¬ 
pergroup; see m, lYB, and the references given there. We may identify Xq/H topologically 
with the alcove Aq via [cost, 2 ;] *pH 1 —>• t. It is then immediate from (|4.ip that the quotient 
convolution on Aq derived from *p is given by 




1 


K. 


V JB^ Jsu{q) 


(4.3) 

/ (arccos (Using (—sin t re sin t' + cost v cost'))) ■ A{Iq — w*w)^~‘^'^dvdw 


for t,t' £ Aq and / G Ch{Aq). These are precisely the hypergroup convolutions studied 
in Section 6 of [RR] . For integers p > 2q, this connection just reflects the fact that for 
G = SU{p + q),K = S{U{p) X U{q)) and Ki = SU{p) x SU{q), we have 


(G//Xi)/(X//Xi)2xG//X 


as hypergroups. This a fact which holds for general commutative double coset hypergroups, 
see Theorem 14.3 of [Jj. 
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4.6 Remarks. 1. We mention at this point that the Haar measure of the hypergroup 
{Xq, *p), which was obtained in Theorem 14.41 bv an orthogonality argument, can alternatively 
be calculated by using Weil’s integration formula for Haar measures on hypergroups and 
their quotients (see [Her ], m)- In the same way as in the non-compact case treated in 
|V3j . the Haar measure can thus be obtained from the known Haar measure of the quotient 
{Xq,*p)/H. For integers p > 2q, the hypergroup {Xq,*p) can be identified with the double 
coset hypergroup SU{p + q)//SU{p) x SU{q), and its Haar measure therefore coincides by 
construction (see with the pushforward measure of the Haar measure on SU{p + q) under 
the canonical projection 

SU{p + q)^ SU{p + q)//SU{p) X SU{q) ~ Xg . 

2. Concerning their structure, the hypergroups {Xq, *p) are also closely related to continuous 
family of commutative hypergroups (Cq x M, *p) with p >2q — l and the HC^-Weyl chamber 
Cq = {(xi,...,Xq) G M”? : > ... > Xq > 0} which were studied in |V2j . For integral 

p, {Cq X M, *p) is an orbit hypergroup under the action of U{p) x U{q) on the Heisenberg 
group Mp^q{C) X M. The characters are given in terms of multivariable Bessel- and Laguerre 
functions. 


5 Continuous product formulas for Heckman-Opdam Jacobi 
polynomials 

Fix the rank q > 1 and a parameter p C\2q — l,oo[. Recall that for I G Z the functions 

1 

(pl i{[cost,z]) = ■ JJcosl'lfj • Rx{k{p,q,l)-,t) 

i=i 

satisfy the product formula of Theorem 14.21 We shall now extend this fomula to exponents I G 
M via Carlson’s theorem, and write it down as a product formula for the Jacobi polynomials 
R\{k]t) with k = k{p,q,l). This will work out smoothly only for non-degenerate arguments 
t, t' G Aq with tiX'i Notice first that for a product formula for the Jacobi polynomials, 

we may restrict our attention to I G [0,oo[ as k{p,q,l) depends on |/| only. In the following, 
we shall use the abbreviation 


d{t,t']v,w) := — sintwsinf + costucost'. 

The main result of this section is 

5.1 Theorem. Let q > 1 be an integer, p G.]2q — l,oo[, I G [0,oo[, and k = k{p,q,l). Then 
for all A G and t,t' G Aq with ti,t'^ / 7r/2, 


R\{k-,t)Rx{k-,t') = — [ I R\{t,aiccos{asing{d{t,t'-,v,w))))- (5.1) 

Jb„ Jsu(q) 

A{Iq - w*w)P-^^dvdw. 


Re 


A{d{t,t'■,v,w)) 
A(cos t) • A(cos t') 
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Proof. For I G Z_|_, z = z' = 1 and t,t' G Ag, the product formula in Theorem 14.21 implies 


(rj costj costj) ■ R\{k-,t)R\{k;t') = 
i=i 


(5.2) 


/ / A{d{t,t']v,w)y ■ R\(^k]avccos{asing{d{t,t''iV,w)))) ■ A{Iq — w*wy ‘^^dvdw. 

JBa Jsu{q) 


Our condition on t, t' assures that both sides of (|5.2I) are analytic in I for Re(t) > 0. Moreover, 
by Section 11 of [M]j the coefficients of the Jacobi polynomials R\{k-,t) with respect to the 
exponentials (// G iJf) are rational in k. Therefore both sides of (15.21) satisfy the growth 
condition of Carlson’s theorem. This implies that (15.2p remains correct for all I G [0,oo[. As 
R\{k', t) is real for all A G T+ and t £ Ag, the claimed product formula now follows from (15.211 
by taking real parts. 

□ 


Contrary to the non-compact case in Section 5 of [V3j . it seems to be difficult in the 
present setting to derive positivity of the product formula (|5.1I) except for the known case 
I = 0. This problem already appears in rank one, i.e. for q = 1. We discuss this case for 
illustration. 

5.2 Example. Let q = 1, p > 2q — 1 = 1, I £ [0, oo[, and k = k{p, q,l) = {p — 1 — I, ^ + 1). 
Resuming the notions from Example 13.61 we have 


a = -|- A;2 — 1/2 = p — 1 > 0, (3 = k2 — 1/2 = 1. 

Consider the classical (normalized) one-dimensional Jacobi polynomials R^'^^ with 

Ri“’«(cos20) = R 2 n(fc; 0 ), 

c.f. (13.9p . With the identity cos 20 = 2cos^ 0 — 1, product formula (|5.ip becomes 

(cos 20)Ri"’^) (cos 20') = (5.3) 


a y r 
Jo J—TT 


Rt^\2\b + - 1 ) • . ^(1 _ yy-^drdif 


for 0,0' G [0,7r/2[, with a := sin0 sin0' and b := cos 0 cos 0' > 0. Following Section 5 of 
we use the change of variables (r, (p) i—>■ {t, ip) with 

b + are^'^ = and t > 0. 

Then a?r dr dp = t dt dip and identity (15.31) becomes, for 0 < 0, 0' < 7r/2: 

Ri"’^)(cos20)Ri“’^)(cos20') = (5.4) 

rOO PTT 

/ / R^y'^\2y — 1) — b"^ — y + 2bt cos ip)y~^t dt dip 

Jo J-TT 

R^“’^)(2t2 - 1)^ -b^ -y + 2btcosip)y-^dipy^^^dt. 


a 1 

TT bda‘^°‘ 
a 2 

TT bda?'^ 


Here the notation 




if a: > 0, 
0 if X < 0 
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is used. Notice for the last equality that t = \b + ore**^! < a + 6 < 1. Notice also that for 
6 = 0 ov 0' = 0, the product formula degenerates in a trivial way due to = 1. 

On the other hand, for a > /3, the Jacobi polynomials satisfy the well-known positive 
product formula im) 

(cos (cos 20') = (5.5) 

Rt^\2\b + - 1 ) • {I - ip dr dip 

with some constant Ca^p > 0. By the same substitution as above this can be brought into 
kernel form, 

(cos (cos 20') = (5.6) 

= - l)(^ {a‘^ - b'^ - ^ + 2btcosil})%~^~^sm^^'iljd^y‘^^+^dt. 

As the integrals in (15.4p and ()5.6p are identical for all n, we conclude that, for all t G [0,1], 




J {a‘^-b'^-t^+2btcos'ip)l-^-^ sm^^^pd^|: = ‘^ J {a‘^-b‘^-t‘^+2bt cos ^|J)'^-^d^p. 

(5.7) 

This identity seems not obvious, and it would be desirable to have an elementary proof for 
it which possibly might be extended to the higher rank case. 


So far, a positive product formula such as formula (j5.5p of Koornwinder seems to be a 
difficult task in rank q > 2. However, at least the first step above in the case q = 1, that is 
from (15. jp to (j5.4p . can be partially extended to g > 2. Indeed, consider the product formula 
m for t,t' G Aq. We define the matrices 

ai := sint, 02 := sint' , b = b{v) := costv cost' G Mq{C) 


and consider the polar decomposition b — aiwa2 ='■ ru with positive semidefinite r G Mq{C) 
and u G U{q). We now carry out the change of variables in two steps. First, for w := 
aiwa2 G Mq{C), we have dw = const ■ A(aia 2 )^'^ dw. Moreover, the polar decomposition 
b — w = y/ru (for non-singular r) leads to dw = const ■ drdu, where dr means integration 
with respect to the Lebesgue measure on the cone Hg of positive definite matrices as an open 
subset of the vector space of all Hermitian matrices, and du is the normalized Haar measure 
of U{q); see Proposition XVI.2.1 of m- Formula (15. ip now reads 


Rx{t,t)Rx{k;t') = 


= const 


Rx{k] 


arccos (7 


sing 


iVr))) 


A(r)^/^ 


SU{q) JU{q) 


A(cost)^A(cost')^A(sint)2'?A(sint')^'? 
A(^H{t, t', r, u, u)+) A(u)^ dvdu) dr (5-8) 


where 

H{t,t' ,r,u,v) = Iq — a 2 ^ {b{v)* — u* ^/r) [b{v) — x/ru) a 2 ^ 

and the subscript + means that this term is put zero for matrices which are not positive 
definite. The analysis of the origin of these formulas shows that in the outer integral, r in 
fact runs through the set {r G fig : 7g — r > 0}. One may speculate that (|5.8p can be 
brought into a kernel form by replacing the integration over r G Og by an integration over 
the spectrum of r as a subset of {p G M”? : 1 > pi > • • • > Pg > 0}. 
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